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ABSTRACT 

The purpose of this study was to determine how 
cognitive style instructions to respondents of a survey instrument 
affected the resulting psychometric properties of the scale. It was 
hypothesized that a group which is instructed to carefully respond 
will have a mean score further from the neutral point, a larger 
standard deviation, and a higher internal consistency, and will take 
more time in responding, than will a group which is instructed to 
immediately respond. Using an instrument designed to measure feelings 
about the manipulation of people in interpersonal situations, 
respondents were asked to either "carefully respond" or "immediately 
respond" to each item. The results suggested that neither the central 
tendency, dispersion, nor the reliability of the scores were strongly 
affected by type of directions. The amount of time spent in 
responding was only slightly affected. (DWH) 
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The purpose of this study was to determine how cognitive style Instructions 
to respondents of a survey instrument affected the resulting psychometric 
properties of the scale. Using an Instrument designed to measure feelings 
about the manipulation of people in interpersonal situations, respondents 
were asked to either "carefully respond" or "immediately respond" to each 
item. The results suggested that neither the central tendency, dispersion, 
nor reliability of the scores were strongly affected by type of directions. 
The amount of time spent in responding was only slightly affected. 
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The Effects of Type of Instructions on the 
Psychometric Properties of a Survey Instrument 



In many educational applications of survey research, students are 
requested to respond to affective Instruments. The accompanying directions 
for such Instruments vary widely in the amount of reflection requested of the 
respondent. Some Instructions call for immediate responses, some encourage 
careful reflection, and others omit any reference to the matter. 

Three examples will illustrate some of the diversity 1n directions 
» commonly used with affective instruments (from Robinson and Shaver, 19.73): 

"First impressions are usually best in such situations" 
(p. 593). 

"Circle the response which best represents your immediate 
reaction to the opinion expressed" (p. 530). 

"It is impor nt that you try to answer each item frankly 
and honestly" (p. 121). 

Because of the widespread use of such instruments in survey research, it 
would appear that the proper choice of instructions would be a matter of 
concern to researchers. This study was thus designed to empirically Investigate 
how Instructions to carefully respond and instructions to Immediately respond 
would differ in their effects upon certain psychometric properties. 

A closely related phenomenon has long been investigated in the area of 
cognitive test construction; namely, the question as to whether examinees 
should feel free to change their responses to objective test Items or depend 
solely upon their first impressions. Most investigators have focused their 
attention toward the effects of answer changing on total scores, with the 
reasonably consistent finding that response changes improve scores more often 
than not (Mueller and Wasser, 1977). Crocker and Benson (1980) reported that 
answer changing produced virtually no change in the internal consistency of 
the items. 

For survey research using affective scales, it appears reasonably analogous 
to expect that subjects who employ careful reflection will produce less random 
error than those who respond immediately. Therefore* 1n this study 1t was 
hypothesized that a group which 1s instructed to carefully respond will have 
a mean score further from the neutral point, a larger standard deviation, and 
a higher Internal consistency, and will taka more time in responding, than will 
a group which 1s instructed to Immediately respond. 

Method 

Subjects 

To maximize the probability that the treatments would actually be 
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Implemented by the subjects, a sample of mature students was sought who would 
be capable of following the prescribed directions. It was also desired to 
select a sample sufficiently homogeneous that one instrument would be 
appropriate, yet diverse enough to reflect the general popjlation of prospective 
teachers. Such a sample was selected by using the pupils enrolled in a required, 
third year educational measurement course at a midwestern university. The 
group consisted almost exclusively of sophomores and juniors. A total of 
150 subjects in seven classrooms participated. 

Materials 

Since the sample was comprised of prospective teachers, an Instrument 
was sought which would measure a construct important to the teaching act, 
particularly one dealing with the role of influencing and directing students. 
The Machavel lianism scale (Robinson and Shaver, 1973] was selected because it 
appeared to satisfy this condition, inasmuch as it purports "to tap ... the 
degree to which a person feels other people are manipulable in interpersonal 
situations" (p. 590). The following is an illustrative item: "The best way 
to handle people is to tell them what they want to hear." The scale consists 
of 20 items in a Likert type format. In the original format, a seven-point 
scale was used for each item, but for this study, the instrument was modified 
by using the more conventional five-point scale. 

Two forms of instructions were prepared, worded as follows: 

"Carefully Respond" instructions: "Consider each question carefully . 
Be as sincere and accurate as possible within the limited time. If upon 
reflection, you desire to alter your response, please feel free to change 
it by erasing or marking it out and circling another response." 

"Immediately Respond" instructions: "Indicate your immediate reaction 
without thinking too much about it. Remember that first impression s are 
usually best in such matters. Go rapidly; do not go back once youlTave marked 
a statement." 

Procedure 

Within each of the seven classrooms, students were invited to voluntarily 
and annonymously respond to the instrument. As far as could be determined, 
only one or two chose not to participate. Copies of the instrument containing 
either "Carefully Respond" or "Immediately Respond" instructions were randomly 
distributed. The students were then requested to read and follow the 
instructions, and to turn in their papers immediately upon completion. 

As the papers were turned in, .they were numbered sequentially, to allow 
for a time analysis. 

4 

To analyze the scale data, the response positions (strongly disagree 
through strongly agree) were assigned the integers one through five, respectively, 
thus allowing for the total scores to potentially range from 2C to 100 points. 
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■ Results 

The medn scores and standard deviations for both groups are shown in 
Table 1. When the mean scores were compared, the resulting value of t = .18 
Indicated that no significant difference existed. When the standard deviations 
were compared, the resulting value of F * 1.15 (df = 74, 73; p > .25) also 
indicated no significant difference. 

The reliability estimates for both groups are shown in Table 1 as alpha 
values. When these values were compared by conceptual izaing them as correlation 
coefficients, the resulting value of z * .83 likewise indicated no significant 
difference (Hays, 1973). 

Since the students turned in their papers as soon as they were finished, 
it was possible to assign ranks representing the amount of elapsed time. The 
data within each class were analyzed with the Mann-Whitney U-test, and the 
results are shown in Table 2. Using the Stouffer method detailed by Mosteller 
and Bush (1954, p. 329) for combining the results of statistical tests, it 
was found that the groups differed in the expected direction at a statistical 
significant level (z * 3.46, p < .01). However, an examination of Table 2 
suggests that the effect c f the directions on the working speed of the subjects 
is probably small and unlikely to be noticeable in classes of less than 
30 pupils. 

Discussion 

For the survey researcher, one of the implications of this study appears 
to be that, since subjects tend to give little heed to the instructions on 
how much time or attention to give to each item, it might be best to just ask 
the subject for their response, without giving any cognitive style instructions. 
Another implication may be that, when reading reports of survey research, 
the reader can probably correctly assume that the obtained results were not 
seriously affected by the type of cognitive style instructions to the subjects. 

It may be appropriate to conclude with an ethical consideration. One of 
the hailmarks of a civilized society is that the members carefully consider 
their thoughts before translating them into action. Although immediate 
reactions may be improved with training, it appears doubtful that they will 
ever be better than carefully reasoned ones. Therefore, in most educational 
settings, it would appear to be advisable for survey researchers to ask the * 
respondents to give their best reasoned judgments within a moderate ,ime 
period. Interpreters of the results will then be able to have justifiable 
confidence in the data, both from psychometric and ethical viewpoints. 
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Table 1 

Means, Standard Deviations and Reliabilities for the Machiavellianism Scale 



Group N Mean SD Alpha 

"Carefully Respond" 7S 66.48 7.67 .736 

"Immediately Respond" 74 66.26 7.16 .665 



Table 2 

The Mann-Whitney U Test Analysis of the Amount of Time Spent 



Class 


Carefully Respond 
n R 


Immediately Respond 
n R 


z 


1 


11 


145 


11 


108 


1.15 


2 


13 


209 


13 


142 


1.67 


3 


8 


82 


7 


38 


1.97 


4 


8 


80 


8 


-56 


1.16 


5 


13 


210 


13 


141 


1.72 


6 


11 


135 


10 


96 


.92 


7 


12 


161 


12 


139 


.58 
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